Repository navigation
scripts/data-inventory.py: a dataset manifest without a single value - #184
Merged
Merged
Conversation
…ingle value Path, size, SHA-256 and, for tables, rows, column names and the columns whose names suggest personal data. CSV/TSV via the standard library, with names withheld when a first row looks like data; .dta/.sav/.sas7bdat via pyreadstat metadata-only and Parquet via its footer when installed. /replication-package starts its dataset manifest from it, and on restricted data the author runs it and shares only the output. Four portability cases (cp1252, non-ASCII names, hash, missing path); a value-leak mutant and a header mutant each fail their case (LEDGER row). Idea from the World Bank's reproducibility skill; independent code.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b96ec487e8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… manifest needs clearance on restricted data Codex review on PR #184 (two P1s): - A headerless CSV whose first record is all text (Alice,London) was listed as column names. No rule can tell it from a header, so first rows are listed only on --header, the user's check that every file given has one; a numeric first row is still withheld. - On restricted data the instruction cleared only column names, though file names, sizes and exact row counts are derived from the data too. The script, /replication-package and the CHANGELOG now require the whole manifest to go through the agreement's disclosure review, and --no-counts drops the row counts. Six portability cases; three mutants each fail their own (LEDGER row updated).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scripts/data-inventory.pylists every file under the given paths (defaultdata/raw): path, size, SHA-256 and, for tables, row count, column names, and the columns whose names suggest personal data. It never outputs a value.--header, your check that every file given has a header row. No rule can tellAlice,Londonfrom a header. Even with--header, a numeric first row is withheld..dta,.sav,.sas7bdat,.xpt: read with pyreadstat's metadata-only mode. Parquet: read from its footer. Both only when the package is installed; otherwise just the size and hash are listed.--no-countsdrops the row counts./replication-packagebuilds its dataset manifest from it. On restricted data, the whole manifest (file names, sizes, row counts, column names) goes through the agreement's disclosure review before it leaves the machine or reaches a model.Review
Changed defaults
Test plan
--header, no first row is listed, including an all-text one;--no-countsworks;--header. Ledger row added../scripts/backtest.shpasses all 11 gates. Portability suite: 239 cases.